Vision-based retargeting for endoscopic navigation

Endoscopy is a standard procedure for visualising the human gastrointestinal tract. With the advances in biophotonics, imaging techniques such as narrow band imaging, confocal laser endomicroscopy, and optical coherence tomography can be combined with normal endoscopy for assisting the early diagnosis of diseases, such as cancer. In the past decade, optical biopsy has emerged to be an effective tool for tissue analysis, allowing in vivo and in situ assessment of pathological sites with real-time feature-enhanced microscopic images. However, the non-invasive nature of optical biopsy leads to an intra-examination retargeting problem, which is associated with the difficulty of re-localising a biopsied site consistently throughout the whole examination. In addition to intra-examination retargeting, retargeting of a pathological site is even more challenging across examinations, due to tissue deformation and changing tissue morphologies and appearances. The purpose of this thesis is to address both the intra- and inter-examination retargeting problems associated with optical biopsy. We propose a novel vision-based framework for intra-examination retargeting. The proposed framework is based on combining visual tracking and detection with online learning of the appearance of the biopsied site. Furthermore, a novel cascaded detection approach based on random forests and structured support vector machines is developed to achieve efficient retargeting. To cater for reliable inter-examination retargeting, the solution provided in this thesis is achieved by solving an image retrieval problem, for which an online scene association approach is proposed to summarise an endoscopic video collected in the first examination into distinctive scenes. A hashing-based approach is then used to learn the intrinsic representations of these scenes, such that retargeting can be achieved in subsequent examinations by retrieving the relevant images using the learnt representations. For performance evaluation of the proposed frameworks, extensive phantom, ex vivo and in vivo experiments have been conducted, with results demonstrating the robustness and potential clinical values of the methods proposed.Open Acces

A Multi-Robot Cooperation Framework for Sewing Personalized Stent Grafts

This paper presents a multi-robot system for manufacturing personalized medical stent grafts. The proposed system adopts a modular design, which includes: a (personalized) mandrel module, a bimanual sewing module, and a vision module. The mandrel module incorporates the personalized geometry of patients, while the bimanual sewing module adopts a learning-by-demonstration approach to transfer human hand-sewing skills to the robots. The human demonstrations were firstly observed by the vision module and then encoded using a statistical model to generate the reference motion trajectories. During autonomous robot sewing, the vision module plays the role of coordinating multi-robot collaboration. Experiment results show that the robots can adapt to generalized stent designs. The proposed system can also be used for other manipulation tasks, especially for flexible production of customized products and where bimanual or multi-robot cooperation is required.Comment: 10 pages, 12 figures, accepted by IEEE Transactions on Industrial Informatics, Key words: modularity, medical device customization, multi-robot system, robot learning, visual servoing, robot sewin

Inhibition of TRPA1 Attenuates Doxorubicin-Induced Acute Cardiotoxicity by Suppressing Oxidative Stress, the Inflammatory Response, and Endoplasmic Reticulum Stress

The transient receptor potential ankyrin 1 (TRPA1) channel is expressed in cardiomyocytes and involved in many cardiovascular diseases. However, the expression and function of TRPA1 in doxorubicin- (Dox-) induced acute cardiotoxicity have not been elucidated. This study aimed at investigating whether blocking the TRPA1 channel with the specific inhibitor HC-030031 (HC) attenuates Dox-induced cardiac injury. The animals were randomly divided into four groups: control, HC, Dox, and Dox + HC. Echocardiography was used to evaluate cardiac function, and the heart was removed for molecular experiments. The results showed that the expression of TRPA1 was increased in the heart after Dox treatment. Cardiac dysfunction and increased serum CK-MB and LDH levels were induced by Dox, but these effects were attenuated by HC treatment. In addition, HC mitigated Dox-induced oxidative stress, as evidenced by the decreased MDA level and increased GSH level and SOD activity in the Dox + HC group. Meanwhile, HC treatment lowered the levels of the proinflammatory cytokines IL-1β, IL-6, IL-17, and TNF-α induced by Dox. Furthermore, HC treatment mitigated endoplasmic reticulum (ER) stress and cardiomyocyte apoptosis induced by Dox. These results indicated that inhibition of TRPA1 could prevent Dox-induced cardiomyocyte apoptosis in mice by inhibiting oxidative stress, inflammation, and ER stress

Self-Supervised Siamese Learning on Stereo Image Pairs for Depth Estimation in Robotic Surgery

Robotic surgery has become a powerful tool for performing minimally invasive procedures, providing advantages in dexterity, precision, and 3D vision, over traditional surgery. One popular robotic system is the da Vinci surgical platform, which allows preoperative information to be incorporated into live procedures using Augmented Reality (AR). Scene depth estimation is a prerequisite for AR, as accurate registration requires 3D correspondences between preoperative and intraoperative organ models. In the past decade, there has been much progress on depth estimation for surgical scenes, such as using monocular or binocular laparoscopes [1,2]. More recently, advances in deep learning have enabled depth estimation via Convolutional Neural Networks (CNNs) [3], but training requires a large image dataset with ground truth depths. Inspired by [4], we propose a deep learning framework for surgical scene depth estimation using self-supervision for scalable data acquisition. Our framework consists of an autoencoder for depth prediction, and a differentiable spatial transformer for training the autoencoder on stereo image pairs without ground truth depths. Validation was conducted on stereo videos collected in robotic partial nephrectomy.Comment: A two-page short report to be presented at the Hamlyn Symposium on Medical Robotics 2017. An extension of this work is on progres

Enhancing photoelectrochemical CO2 reduction with silicon photonic crystals

The effectiveness of silicon (Si) and silicon-based materials in catalyzing photoelectrochemistry (PEC) CO2 reduction is limited by poor visible light absorption. In this study, we prepared two-dimensional (2D) silicon-based photonic crystals (SiPCs) with circular dielectric pillars arranged in a square array to amplify the absorption of light within the wavelength of approximately 450 nm. By investigating five sets of n + p SiPCs with varying dielectric pillar sizes and periodicity while maintaining consistent filling ratios, our findings showed improved photocurrent densities and a notable shift in product selectivity towards CH4 (around 25% Faradaic Efficiency). Additionally, we integrated platinum nanoparticles, which further enhanced the photocurrent without impacting the enhanced light absorption effect of SiPCs. These results not only validate the crucial role of SiPCs in enhancing light absorption and improving PEC performance but also suggest a promising approach towards efficient and selective PEC CO2 reduction

From Macro to Micro: Autonomous Multiscale Image Fusion for Robotic Surgery

In recent years, minimally invasive robotic surgery has shown great promises for enhancing surgical precision and improving patient outcomes. Despite these advances, intraoperative tissue characterisation (such as the identification of cancerous tissue) still relies on traditional biopsy and histology, a process that is time-consuming and often disrupts the normal surgical workflow. In order to provide effective intra-operative decision-making, emerging optical biopsy techniques, such as probe based confocal laser endomicroscopy (pCLE) and optical coherence tomography (OCT), have been developed to provide real-time in vivo, in situ assessment of tissue micro-structures. Clinical deployment of these techniques, however, requires large area surveillance, from macro (mm/cm) to micro (µm) coverage in order to differentiate underlying tissue structures. This article provides a real-time multi-scale fusion scheme for robotic surgery. It demonstrates how the da Vinci surgical robot, used together with the da Vinci Research Kit, can be used for automated 2D scanning of pCLE/OCT probes, providing large area tissue surveillance by image stitching. Open-loop control of the robot provides insufficient precision for probe scanning, and therefore the motion is visually servoed using the live pCLE images (for lateral position) and OCT images (for axial position). The resulting tissue maps can then be fused in real-time with a stereo reconstruction from the laparoscopic video, providing the surgeon with a multi-scale 3D view of the operating site

Aggregation‐induced emission luminogen: A new perspective in the photo‐degradation of organic pollutants

Both the variety and uniqueness of organic semiconductors has contributed to the rapid development of environmental engineering applications and renewable fuel production, typified by the photodegradation of organic pollutants or water splitting. This paper presents a rare example of an aggregation‐induced emission luminogen as a highly efficient photocatalyst for pollutant decomposition in an environmentally relevant application. Under irradiation, the tetraphenylethene‐based AIEgen (TPE‐Ca) exhibited high photo‐degradation efficiency of up to 98.7% of Rhodamine B (RhB) in aqueous solution. The possible photocatalytic mechanism was studied by electron paramagnetic resonance and X‐ray photoelectron spectroscopy spectra, electrochemistry, thermal imaging technology, ultra‐performance liquid chromatography and high‐definition mass spectrometry, as well as by density functional theory calculations. Among the many diverse AIEgens, this is the first AIEgen to be developed as a photocatalyst for the degradation of organic pollutants. This research will open up new avenues for AIEgens research, particularly for applications of environmental relevance

Towards finger motion tracking and analyses for cardiac surgery

Robot Assisted Surgery is attracting increasing amount of attention as it offers numerous benefits to patients as well as surgeons. Heart surgery requires a high level of precision and dexterity, in contrast to other surgical specialties. Robot assisted heart surgery is not as widely performed due to numerous reasons including a lack of appropriate and intuitive surgical interfaces to control minimally invasive surgical tools. In this paper, finger motion of the surgeon is analyzed during cardiac surgery tasks on an ex-vivo animal model with the purpose of designing a more intuitive master console. First, a custom finger tracking system is developed using IMU sensors, which is lightweight and comfortable enough to allow free movement of the surgeon’s fingers/hands while using instruments. The proposed system tracks finger joint angles and fingertip positions for three involved fingers (thumb, index, middle). Accuracy of the IMU sensors has been evaluated using an optical tracking system (Polaris, NDI). Finger motion of the cardiac surgeon while using a Castroviejo instrument is studied in suturing and knotting scenarios. The results show that PIP and MCP joints have larger Range Of Motion (ROM), and faster rate of change compared to other finger/thumb joints, while thumb has the largest Fingertip WorkSpace (FWS) of all three digits

Predicting disease-associated substitution of a single amino acid by analyzing residue interactions

<p>Abstract</p> <p>Background</p> <p>The rapid accumulation of data on non-synonymous single nucleotide polymorphisms (nsSNPs, also called SAPs) should allow us to further our understanding of the underlying disease-associated mechanisms. Here, we use complex networks to study the role of an amino acid in both local and global structures and determine the extent to which disease-associated and polymorphic SAPs differ in terms of their interactions to other residues.</p> <p>Results</p> <p>We found that SAPs can be well characterized by network topological features. Mutations are probably disease-associated when they occur at a site with a high centrality value and/or high degree value in a protein structure network. We also discovered that study of the neighboring residues around a mutation site can help to determine whether the mutation is disease-related or not. We compiled a dataset from the Swiss-Prot variant pages and constructed a model to predict disease-associated SAPs based on the random forest algorithm. The values of total accuracy and MCC were 83.0% and 0.64, respectively, as determined by 5-fold cross-validation. With an independent dataset, our model achieved a total accuracy of 80.8% and MCC of 0.59, respectively.</p> <p>Conclusions</p> <p>The satisfactory performance suggests that network topological features can be used as quantification measures to determine the importance of a site on a protein, and this approach can complement existing methods for prediction of disease-associated SAPs. Moreover, the use of this method in SAP studies would help to determine the underlying linkage between SAPs and diseases through extensive investigation of mutual interactions between residues.</p

Autonomous scanning for endomicroscopic mosaicing and 3D fusion

Robot-assisted minimally invasive surgery can benefit from the automation of common, repetitive or well-defined but ergonomically difficult tasks. One such task is the scanning of a pick-up endomicroscopy probe over a complex, undulating tissue surface to enhance the effective field-of-view through video mosaicing. In this paper, the da Vinci® surgical robot, through the dVRK framework, is used for autonomous scanning and 2D mosaicing over a user-defined region of interest. To achieve the level of precision required for high quality mosaic generation, which relies on sufficient overlap between consecutive image frames, visual servoing is performed using a combination of a tracking marker attached to the probe and the endomicroscopy images themselves. The resulting sub-millimetre accuracy of the probe motion allows for the generation of large mosaics with minimal intervention from the surgeon. Images are streamed from the endomicroscope and overlaid live onto the surgeons view, while 2D mosaics are generated in real-time, and fused into a 3D stereo reconstruction of the surgical scene, thus providing intuitive visualisation and fusion of the multi-scale images. The system therefore offers significant potential to enhance surgical procedures, by providing the operator with cellular-scale information over a larger area than could typically be achieved by manual scanning